Papers by Yong Jae Lee
MATE: Meet At The Embedding - Connecting Images with Long Texts (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in Vision Language Models (VLMs) focus on aligning images with short descriptive captions. |
| Approach: | They propose a method that combines VLMs with Large Language Models to efficiently align images with long texts without additional text pairs. |
| Outcome: | The proposed method bridges the gap between VLM and LLM without additional image-long text pairs. |
A Visual Attention Grounding Neural Model for Multimodal Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to multimodal machine translation do not integrate visual information into the translation process. |
| Approach: | They propose a multimodal machine translation model that utilizes parallel visual and textual information. |
| Outcome: | The proposed model outperforms existing methods on the Multi30K and Ambiguous COCO datasets. |
CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples (2024.findings-acl)
Copied to clipboard
| Challenge: | Using high-performing text and image generation models, we can improve compositional reasoning capabilities for large multimodal models. |
| Approach: | They propose a framework to improve visio-linguistic compositional reasoning capability for multimodal models by leveraging physical grounding and using image-based models for semantic counterfactual fine-tuning. |
| Outcome: | The proposed framework outperforms GPT-4V and DALLE-3 on a new dataset and checkpoints. |
Agentic Very Long Video Understanding (2026.acl-long)
Copied to clipboard
Aniket Rege, Arka Sadhu, Yuliang Li, Kejie Li, Ramya Korlakai Vinayak, Yuning Chai, Yong Jae Lee, Hyo Jin Kim
| Challenge: | Existing methods for understanding video over long periods of time are limited . eGAgent system provides tools for structured search and reasoning over entity scene graphs . |
| Approach: | They propose a system that can interpret and recall video over days or weeks . they use entity scene graphs to equip a planning agent with tools for structured search and reasoning . |
| Outcome: | The proposed method achieves state-of-the-art performance on EgoLifeQA and Video-MME-long datasets. |
VGBench: A Comprehensive Benchmark of Vector Graphics Understanding and Generation for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current vision models use pixels to rasterize the visual world, but vector graphics are not the best or unique way to represent visual content. |
| Approach: | They propose a benchmark for vector graphics processing with capable Large Language Models . they use a set of questions to evaluate vector graphics formats and a wide range of prompting techniques . |
| Outcome: | The proposed benchmark compares LLMs on rasterized representations with vector graphics . it shows that LLM models show strong capability on both aspects . |